regen.mde 0.2.2

package/LICENSE

@@ -0,0 +1,16 @@
+Apache License
+Version 2.0, January 2004
+
+Copyright 2026 LIFE AI
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

package/README.md

@@ -0,0 +1,295 @@
+# regen.mde
+
+regen.mde is the Windows editor and conversion suite for Markdown, Word, and PowerPoint files. Its `build-corpus` CLI converts `.docx`, `.pptx`, and `.ppt` files to Markdown while preserving the pieces that usually break in generic converters:
+
+- Word OMML equations as KaTeX-readable TeX
+- embedded images as local assets, base64 data URIs, or S3/R2-hosted URLs
+- Markdown tables for simple Word tables
+- HTML table fallback for complex tables
+- headings, lists, links, bold, italic, inline code, and code-style paragraphs
+- PowerPoint slide extraction with slide title detection, table mapping, and repetitive footer suppression
+
+## Install
+
+Python is the native runtime:
+
+```powershell
+pip install build-corpus
+```
+
+The npm package ships the Windows installer plus the conversion CLI:
+
+```powershell
+npm pack regen.mde
+```
+
+Extract the package and run `dist\release\regen.mde-<version>-win-x64-setup.exe` for a normal Windows install. The installer creates Start Menu entries for `regen.mde` and `Uninstall regen.mde`, registers right-click Explorer verbs for `.docx` and `.md`, and removes those entries during uninstall.
+
+The legacy global npm command path is still supported for automation:
+
+```powershell
+npm install -g regen.mde
+```
+
+On Windows, the installer and supported automation paths add right-click Explorer menus for `.docx` and `.md` files under `Life AI`:
+
+- `Life AI -> Open in regen.mde`
+- opens `.md` directly and opens `.docx` by converting it into editable Markdown first
+- `Life AI -> Convert to Markdown`
+- runs `build-corpus "%1" --out-same-dir`
+- writes `.md`, `assets`, and reports beside the source document
+- `Life AI -> Convert to Word`
+- runs `build-corpus "%1" --to word --out-same-dir`
+- writes `.docx` and export report beside the source document
+
+Set `BUILD_CORPUS_SKIP_WINDOWS_MENU=1` before a global npm install if you do not want the Explorer menu.
+Set `BUILD_CORPUS_SKIP_EDITOR=1` before a global npm install if you want the CLI conversion verbs but not the editor open verbs.
+
+To remove the Windows Explorer menus without uninstalling the package:
+
+```powershell
+build-corpus --uninstall-windows
+```
+
+If you uninstall the global npm package, `build-corpus` now removes those Explorer menu entries automatically during uninstall.
+
+For a project-local install, use `npx`:
+
+```powershell
+npm install regen.mde
+npx build-corpus --help
+```
+
+On Windows, if `build-corpus` launches a Python executable and fails with `ModuleNotFoundError`, a stale pip install is shadowing the npm command. Remove it with:
+
+```powershell
+py -3 -m pip uninstall build-corpus
+```
+
+For S3/R2 image upload support:
+
+```powershell
+pip install "build-corpus[s3]"
+```
+
+## Basic Usage
+
+```powershell
+build-corpus input.docx --out out
+build-corpus deck.pptx --out out
+build-corpus input.md --to word --out out
+build-corpus input.md --to word --word-template C:\path\custom.dotx --out out
+regen.mde input.md
+regen.mdeditor input.md
+regen-mdeditor input.md
+build-corpus editor input.md
+build-corpus editor input.docx
+```
+
+## regen.mde
+
+regen.mde is a Windows WebView2 desktop app bundled with the package. It uses the same local Build Corpus conversion engine as the CLI:
+
+- Markdown opens directly.
+- Word and PowerPoint files open by converting into Markdown.
+- Save writes Markdown.
+- Save As writes a new Markdown file.
+- Export DOCX writes Word output through the Markdown-to-Word route.
+
+Build the Windows executable locally:
+
+```powershell
+npm run editor:windows
+```
+
+The executable is written to:
+
+```text
+dist\windows-editor\BuildCorpusEditor.exe
+```
+
+Convert every `.docx` in a folder:
+
+```powershell
+build-corpus ./word-files --out ./markdown
+```
+
+Convert every supported file type in a folder (`.docx`, `.pptx`, `.ppt`):
+
+```powershell
+build-corpus ./source-files --out ./markdown
+```
+
+Write Markdown beside each source document:
+
+```powershell
+build-corpus ./word-files --out-same-dir
+```
+
+## Image Modes
+
+Local asset files, the default:
+
+```powershell
+build-corpus input.docx --images assets
+```
+
+Single-file Markdown with base64 image data URIs:
+
+```powershell
+build-corpus input.docx --images base64
+```
+
+Upload images to S3-compatible storage and write public URLs:
+
+```powershell
+build-corpus input.docx --images s3 --config examples\build-corpus.config.example.json
+```
+
+Cloudflare R2 uses the same `s3` mode. Set `endpoint_url` to:
+
+```text
+https://ACCOUNT_ID.r2.cloudflarestorage.com
+```
+
+## Config
+
+Copy `examples/build-corpus.config.example.json` and edit it for your environment.
+
+```json
+{
+  "conversion": {
+    "equations": "tex",
+    "images": "s3"
+  },
+  "output": {
+    "out": "out",
+    "out_same_dir": false
+  },
+  "s3": {
+    "bucket": "build-corpus-assets",
+    "public_base_url": "https://assets.example.com",
+    "prefix": "knowledge-base",
+    "endpoint_url": "https://ACCOUNT_ID.r2.cloudflarestorage.com",
+    "region_name": "auto",
+    "access_key_id": "%R2_ACCESS_KEY_ID%",
+    "secret_access_key": "%R2_SECRET_ACCESS_KEY%"
+  }
+}
+```
+
+Build Corpus expands environment variables in JSON string values, so credentials do not need to be committed.
+
+### Output Placement
+
+There are two output modes.
+
+Write all converted Markdown into one output tree:
+
+```json
+{
+  "output": {
+    "out": "./markdown",
+    "out_same_dir": false
+  }
+}
+```
+
+Write each `.md`, asset folder, and report beside the source `.docx`:
+
+```json
+{
+  "output": {
+    "out_same_dir": true
+  }
+}
+```
+
+The same-dir mode is equivalent to:
+
+```powershell
+build-corpus ./word-files --out-same-dir
+```
+
+## Markdown to Word Templates
+
+Markdown -> Word conversion uses this template precedence:
+
+1. `--word-template <path>`
+2. `word.template` in the JSON config
+3. the bundled installed package template
+4. built-in fallback styles if no template can be found
+
+Template files are treated as style sources. Build Corpus creates a fresh output document body, then applies the template's Word styles, numbering, theme, fonts, and settings. It does not reuse the template body content as the exported document.
+
+## Equations
+
+The default equation mode is parseable TeX:
+
+```powershell
+build-corpus input.docx --equations tex
+```
+
+Equation images are only for visual debugging:
+
+```powershell
+build-corpus input.docx --equations image
+```
+
+## PowerPoint Notes
+
+- `.pptx` is processed directly.
+- `.ppt` is converted to `.pptx` first using LibreOffice (`soffice --headless --convert-to pptx`).
+- Repeated boilerplate blocks that appear on most slides are removed from the emitted Markdown.
+- Slide images are exported from the original package binaries (`ppt/media/*`), not screen-captured display rasters.
+- Markdown output uses size-aware HTML image tags (`<img ... width= height=>`) based on OOXML display extents (`a:xfrm/a:ext`).
+- The export report includes `low_dpi_images` to flag images whose effective on-slide DPI is under 150.
+
+## Validation
+
+The package includes a KaTeX validator for emitted Markdown math:
+
+```powershell
+build-corpus-katex out
+```
+
+## Repeatable Test Wrappers
+
+Run a single known DOCX through conversion plus validators:
+
+```powershell
+.\scripts\run-smoke.ps1 -Docx ".\fixtures\sample.docx" -Out ".tmp\smoke" -Images assets
+```
+
+Run a whole folder corpus:
+
+```powershell
+.\scripts\run-corpus.ps1 -Source ".\fixtures\wordtest" -Out ".tmp\wordtest" -Images base64
+```
+
+Build a public online DOCX corpus for regression testing:
+
+```powershell
+python .\tools\collect_online_docx_corpus.py --out ".tmp\online-docx\source-docx" --target 50
+.\scripts\run-corpus.ps1 -Source ".tmp\online-docx\source-docx" -Out ".tmp\online-docx\markdown"
+```
+
+Build a public online PPTX corpus and compare input/output extraction:
+
+```powershell
+python .\tools\collect_online_pptx_corpus.py --out ".tmp\online-pptx\source-pptx" --target 20
+.\scripts\run-corpus.ps1 -Source ".tmp\online-pptx\source-pptx" -Out ".tmp\online-pptx\markdown"
+python .\tools\compare_pptx_inputs_outputs.py --manifest ".tmp\online-pptx\source-pptx\online-pptx-manifest.json" --out ".tmp\online-pptx\markdown" --report ".tmp\online-pptx\markdown\pptx-io-compare.json"
+```
+
+## Failed Documents
+
+If a document does not convert correctly, open an issue with:
+
+- the `.docx` file if it is safe to share
+- the generated `.md`
+- the `export-report.json`
+- the command and config used
+- a screenshot of the expected Word output if layout is the issue
+
+For confidential files, strip or replace sensitive content before sharing. The useful part is the broken DOCX structure, not the private text.

package/bin/build-corpus-editor.js

@@ -0,0 +1,81 @@
+#!/usr/bin/env node
+const path = require("node:path");
+const { spawn } = require("node:child_process");
+const fs = require("node:fs");
+
+const root = path.resolve(__dirname, "..");
+const callerCwd = process.cwd();
+const candidates = [
+  path.join(root, "dist", "windows-editor", "BuildCorpusEditor.exe"),
+  path.join(root, "desktop", "BuildCorpusEditor", "bin", "Release", "net8.0-windows", "win-x64", "BuildCorpusEditor.exe"),
+];
+const exe = candidates.find((candidate) => fs.existsSync(candidate)) || candidates[0];
+const args = process.argv.slice(2);
+
+if (args.includes("--help") || args.includes("-h")) {
+  console.log(`regen.mde
+
+Usage:
+  regen.mde [document.md|document.docx] [options]
+  regen.mdeditor [document.md|document.docx] [options]
+  regen-mdeditor [document.md|document.docx] [options]
+  build-corpus-editor [document.md|document.docx] [options]
+
+Options:
+  --foreground, --visible       Show the editor as an attached foreground process.
+  --background                  Launch hidden/offscreen for smoke checks.
+  --self-test                   Verify the editor bridge can open the document.
+  --document-self-test          Open, edit, save Markdown, export Word, and reconvert.
+  --smoke-ui                    Run the hidden UI smoke test.
+  --out <dir>                   Output directory for document self-test.
+  --help, -h                    Show this help.
+`);
+  process.exit(0);
+}
+
+const visible = args.includes("--foreground") || args.includes("--visible");
+const background = args.includes("--background") || args.includes("--smoke-ui");
+const launchArgs = normalizeLaunchArgs(args.filter((arg) => arg !== "--foreground" && arg !== "--visible"));
+
+const child = spawn(exe, launchArgs, {
+  cwd: root,
+  stdio: "inherit",
+  windowsHide: background,
+  detached: background && !visible,
+  env: {
+    ...process.env,
+    BUILD_CORPUS_ROOT: root,
+  },
+});
+
+child.on("error", (error) => {
+  console.error(`Build Corpus Editor executable not found: ${exe}`);
+  console.error("Build it with: npm run editor:windows");
+  console.error(error.message);
+  process.exit(1);
+});
+
+child.on("exit", (code) => process.exit(code ?? 0));
+
+function normalizeLaunchArgs(rawArgs) {
+  const normalized = [];
+  for (let index = 0; index < rawArgs.length; index += 1) {
+    const arg = rawArgs[index];
+    if (arg === "--out" && index < rawArgs.length - 1) {
+      normalized.push(arg, resolveFromCaller(rawArgs[index + 1]));
+      index += 1;
+      continue;
+    }
+    if (index === 0 && !arg.startsWith("--")) {
+      normalized.push(resolveFromCaller(arg));
+      continue;
+    }
+    normalized.push(arg);
+  }
+  return normalized;
+}
+
+function resolveFromCaller(value) {
+  if (!value || path.isAbsolute(value)) return value;
+  return path.resolve(callerCwd, value);
+}

package/bin/build-corpus.js

@@ -0,0 +1,41 @@
+#!/usr/bin/env node
+const { spawnSync } = require("node:child_process");
+const path = require("node:path");
+
+const root = path.resolve(__dirname, "..");
+const candidates = process.platform === "win32" ? ["py", "python"] : ["python3", "python"];
+const env = { ...process.env };
+env.PYTHONPATH = env.PYTHONPATH ? `${path.join(root, "src")}${path.delimiter}${env.PYTHONPATH}` : path.join(root, "src");
+const args = process.argv.slice(2);
+
+if (args[0] === "editor") {
+  const editor = path.join(root, "bin", "build-corpus-editor.js");
+  const result = spawnSync(process.execPath, [editor, ...args.slice(1)], {
+    stdio: "inherit",
+    cwd: root,
+    env: { ...env, BUILD_CORPUS_ROOT: root },
+  });
+  process.exit(result.status ?? 1);
+}
+
+if (process.platform === "win32" && args.includes("--uninstall-windows")) {
+  const result = spawnSync(process.execPath, [path.join(root, "bin", "postinstall.js"), "--uninstall"], {
+    stdio: "inherit",
+    cwd: root,
+    env,
+  });
+  process.exit(result.status ?? 1);
+}
+
+let result = null;
+for (const candidate of candidates) {
+  const args = candidate === "py"
+    ? ["-3", "-m", "build_corpus.exporter", ...process.argv.slice(2)]
+    : ["-m", "build_corpus.exporter", ...process.argv.slice(2)];
+  result = spawnSync(candidate, args, { stdio: "inherit", cwd: root, env });
+  if (result.error && result.error.code === "ENOENT") continue;
+  process.exit(result.status ?? 1);
+}
+
+console.error("Build Corpus requires Python 3.10+ on PATH.");
+process.exit(1);

package/bin/postinstall.js

@@ -0,0 +1,187 @@
+#!/usr/bin/env node
+const { spawnSync } = require("node:child_process");
+const path = require("node:path");
+
+const UNINSTALL_MODE = process.argv.includes("--uninstall");
+const SKIP_PIP_INSTALL = process.env.BUILD_CORPUS_SKIP_PIP_INSTALL === "1";
+
+const packageRoot = path.resolve(__dirname, "..");
+const candidates = process.platform === "win32" ? ["py", "python"] : ["python3", "python"];
+const requirements = path.join(packageRoot, "requirements.txt");
+
+function isGlobalInstall() {
+  return process.env.npm_config_global === "true";
+}
+
+function printUsage() {
+  const isGlobal = isGlobalInstall();
+  if (isGlobal) {
+    console.log("Build Corpus installed globally. Run: build-corpus --help");
+    return;
+  }
+
+  console.log("Build Corpus installed locally.");
+  console.log("Run with: npx build-corpus --help");
+}
+
+function warnOnShadowedWindowsCommand() {
+  if (process.platform !== "win32") return;
+
+  const whereResult = spawnSync("where.exe", ["build-corpus"], { encoding: "utf8" });
+  if (whereResult.error || whereResult.status !== 0) return;
+
+  const matches = whereResult.stdout
+    .split(/\r?\n/)
+    .map((line) => line.trim())
+    .filter(Boolean);
+  if (matches.length === 0) return;
+
+  const firstMatch = matches[0].toLowerCase();
+  if (!firstMatch.includes("python") || !firstMatch.endsWith("build-corpus.exe")) return;
+
+  console.warn("");
+  console.warn("WARNING: 'build-corpus' currently resolves to a Python executable:");
+  console.warn(`  ${matches[0]}`);
+  console.warn("That usually means a stale pip install is shadowing the npm command.");
+  console.warn("Fix it with: py -3 -m pip uninstall build-corpus");
+  console.warn("Then rerun your npm install or use: npx build-corpus --help");
+}
+
+function installWindowsContextMenu() {
+  if (process.platform !== "win32") return;
+  if (!isGlobalInstall()) return;
+  if (process.env.BUILD_CORPUS_SKIP_WINDOWS_MENU === "1") return;
+
+  const prefix = process.env.npm_config_prefix || path.join(process.env.APPDATA || "", "npm");
+  const commandPath = path.join(prefix, "build-corpus.cmd");
+  const editorPath = path.join(prefix, "build-corpus-editor.cmd");
+  const nativeEditorPath = path.join(packageRoot, "dist", "windows-editor", "BuildCorpusEditor.exe");
+  const editorCommandPath = fileExists(nativeEditorPath) ? nativeEditorPath : editorPath;
+  const menus = [
+    {
+      keyName: "BuildCorpusToMarkdown",
+      label: "Convert to Markdown",
+      commandValue: `"${commandPath}" "%1" --out-same-dir`,
+      extension: ".docx",
+    },
+    ...(process.env.BUILD_CORPUS_SKIP_EDITOR === "1" ? [] : [
+      {
+        keyName: "BuildCorpusOpenEditor",
+        label: "Open in regen.mde",
+        commandValue: `"${editorCommandPath}" "%1"`,
+        extension: ".docx",
+      },
+      {
+        keyName: "BuildCorpusOpenEditor",
+        label: "Open in regen.mde",
+        commandValue: `"${editorCommandPath}" "%1"`,
+        extension: ".md",
+      },
+    ]),
+    {
+      keyName: "BuildCorpusToWord",
+      label: "Convert to Word",
+      commandValue: `"${commandPath}" "%1" --to word --out-same-dir`,
+      extension: ".md",
+    },
+  ];
+
+  for (const menu of menus) {
+    const commands = [
+      ["delete", `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\Life AI`, "/f"],
+      ["delete", `HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\CommandStore\\shell\\LifeAI.${menu.keyName}`, "/f"],
+      ["delete", `HKCU\\Software\\LifeAI\\ShellCommands\\${menu.keyName}`, "/f"],
+      ["add", `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\${menu.keyName}`, "/v", "MUIVerb", "/d", menu.label, "/f"],
+      ["add", `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\${menu.keyName}`, "/v", "Icon", "/t", "REG_EXPAND_SZ", "/d", menu.keyName === "BuildCorpusOpenEditor" ? editorCommandPath : commandPath, "/f"],
+      ["add", `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\${menu.keyName}\\command`, "/ve", "/d", menu.commandValue, "/f"],
+    ];
+
+    for (const args of commands) {
+      const result = spawnSync("reg.exe", args, { stdio: "inherit" });
+      const status = result.status ?? 1;
+      if (args[0] === "delete" && status === 1 && !result.error) {
+        continue;
+      }
+      if (result.error || status !== 0) {
+        console.warn(`Build Corpus could not add the Windows ${menu.extension} context menu.`);
+        return;
+      }
+    }
+  }
+
+  console.log("Windows Explorer menus added for .docx and .md");
+}
+
+function fileExists(candidate) {
+  try {
+    return require("node:fs").existsSync(candidate);
+  } catch {
+    return false;
+  }
+}
+
+function uninstallWindowsContextMenu() {
+  if (process.platform !== "win32") return 0;
+
+  const menus = [
+    { extension: ".docx", keyName: "BuildCorpusToMarkdown" },
+    { extension: ".docx", keyName: "BuildCorpusOpenEditor" },
+    { extension: ".md", keyName: "BuildCorpusOpenEditor" },
+    { extension: ".md", keyName: "BuildCorpusToWord" },
+  ];
+
+  let exitCode = 0;
+  for (const menu of menus) {
+    const removals = [
+      `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\BuildCorpus`,
+      `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\${menu.keyName}`,
+      `HKCU\\Software\\Classes\\SystemFileAssociations\\${menu.extension}\\shell\\Life AI`,
+      `HKCU\\Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\CommandStore\\shell\\LifeAI.${menu.keyName}`,
+      `HKCU\\Software\\LifeAI\\ShellCommands\\${menu.keyName}`,
+    ];
+
+    for (const key of removals) {
+      const result = spawnSync("reg.exe", ["delete", key, "/f"], { stdio: "inherit" });
+      const status = result.status ?? 1;
+      if (result.error || (status !== 0 && status !== 1)) {
+        console.warn(`Build Corpus could not remove the Windows ${menu.extension} context menu.`);
+        exitCode = 1;
+      }
+    }
+  }
+
+  console.log("Windows Explorer menus removed for .docx and .md");
+  return exitCode;
+}
+
+if (UNINSTALL_MODE) {
+  process.exit(uninstallWindowsContextMenu());
+}
+
+if (SKIP_PIP_INSTALL) {
+  printUsage();
+  installWindowsContextMenu();
+  warnOnShadowedWindowsCommand();
+  process.exit(0);
+}
+
+for (const candidate of candidates) {
+  const args = candidate === "py"
+    ? ["-3", "-m", "pip", "install", "-r", requirements]
+    : ["-m", "pip", "install", "-r", requirements];
+  const result = spawnSync(candidate, args, { stdio: "inherit" });
+  if (result.error && result.error.code === "ENOENT") continue;
+  if ((result.status ?? 1) !== 0) {
+    process.exit(result.status ?? 1);
+  }
+
+  printUsage();
+  installWindowsContextMenu();
+  warnOnShadowedWindowsCommand();
+  process.exit(0);
+}
+
+console.warn("Build Corpus could not find Python to install Python dependencies.");
+printUsage();
+installWindowsContextMenu();
+process.exit(0);

package/bin/regen-mdeditor-install.js

@@ -0,0 +1,27 @@
+#!/usr/bin/env node
+const { spawnSync } = require("node:child_process");
+const path = require("node:path");
+
+if (process.platform !== "win32") {
+  console.error("regen.mde installer is only available on Windows.");
+  process.exit(1);
+}
+
+const root = path.resolve(__dirname, "..");
+const script = path.join(root, "installer", "install-regen-mde.ps1");
+const result = spawnSync("powershell.exe", [
+  "-NoProfile",
+  "-ExecutionPolicy",
+  "Bypass",
+  "-File",
+  script,
+  "-PackageRoot",
+  root,
+  ...process.argv.slice(2),
+], {
+  stdio: "inherit",
+  cwd: root,
+  windowsHide: true,
+});
+
+process.exit(result.status ?? 1);

package/bin/regen-mdeditor-uninstall.js

@@ -0,0 +1,19 @@
+#!/usr/bin/env node
+const { spawnSync } = require("node:child_process");
+const path = require("node:path");
+
+if (process.platform !== "win32") {
+  console.error("regen.mde uninstaller is only available on Windows.");
+  process.exit(1);
+}
+
+const root = path.resolve(__dirname, "..");
+const result = spawnSync(process.execPath, [
+  path.join(root, "bin", "postinstall.js"),
+  "--uninstall",
+], {
+  stdio: "inherit",
+  cwd: root,
+});
+
+process.exit(result.status ?? 1);